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Abstract — This  paper  presents  the  lessoned  learned  from  the 
Joint  Battlespace  Dynamic  Deconfliction  (JBD2)  test  conducted 
in  August  2008.  The  lessons  learned  presented  are  those  collected 
by  the  Test  Integration  Working  Group  (TIWG).  The  JBD2  test 
event  was  executed  in  a  Joint  environment  across  16  different  test 
sites  with  live,  virtual,  and  constructive  elements  connected 
through  the  Joint  Mission  Environment  Test  Capability 
(JMETC)  Virtual  Private  Network  (VPN).  All  Wide  Area 
Network  (WAN)  simulation  traffic  was  exchanged  using  the 
Testing  and  Training  Enabling  Architecture  (TENA). 
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1.  OVERVEIW  OF  JBD2 

DoD  Strategic  Planning  Guidance  demanded  the  creation  of  a 
Joint  environment  testing  capability.  This  demand  led  to  the 
creation  of  the  Joint  Test  and  Evaluation  Methodology 
(JTEM)  project  and  Joint  Mission  Environment  Test 
Capability  (JMETC)  as  part  of  the  larger  testing  in  a  Joint 
environment  initiative.  Concurrently,  the  Combined  Test 
Organization  (CTO)  for  Future  Combat  Systems  (ECS) 
recognized  that  testing  systems  to  meet  Joint  Mission 
Environment  (JME)  requirements  presents  new  challenges  that 
require  new  test  capabilities.  As  the  ECS  testing  strategy 
matured,  along  with  JTEM  and  JMETC,  a  mutually  beneficial 
relationship  was  realized  and  evolved  into  the  Joint 
Battlespace  Dynamic  Deconfliction  (JBD2)  to  provide  the 
following  key  benefits: 

•  For  JTEM,  JBD2  serves  as  a  Joint  capability  “use  case” 
in  order  to  evaluate  the  effectiveness  and  suitability  of 
the  Capability  Test  Methodology  (CTM)  methods  and 
processes  (M&P)  for  designing  and  executing  tests  of 
System  of  Systems  (SoS)  in  the  JME,  utilizing  the  CTM 
v2.0. 

•  For  ECS  CTO,  JBD2  will  establish  a  rigorous  test 
context  to  examine  ECS  test  technology  capabilities 
needed  for  testing  in  a  JME  in  support  of  ECS 
Milestone  (MS)  C  test  activities. 


•  For  JMETC,  JBD2  will  mature  the  baseline  capability 
to  support  SoS  level  testing  across  the  Joint  Services 
and  characterize  the  network  infrastructure. 

The  JBD2  test  event  provided  a  high  fidelity,  real-time, 
rapidly  configurable,  distributed  network  including  virtual  and 
constructive  models  linked  with  live  systems.  The  purpose  of 
the  environment  is  to  evaluate  command  and  control  (C2)  for 
Joint  Fires  (JFIRES)  and  Joint  Close  Air  Support  (JCAS)  as 
well  as  support  the  development  and  testing  initiatives  for  the 
partnered  organizations.  The  JBD2  test  was  executed  August 
4-7,  2008. 

A.  Roles  of  the  Working  Groups 

There  are  many  functions  and  duties  required  in  the  design  and 
implementation  of  an  operationally-relevant  Live,  Virtual,  and 
Constructive  (LVC)  test  environment.  For  JBD2  these  duties 
along  with  the  test  execution  and  data  analysis  functions  were 
divided  between  four  working  groups: 

•  Operational  Capabilities  Working  Group  (OCWG)  - 

Define  tactically  relevant  scenarios  for  JBD2  Joint 
Operational  Context  for  Test  (JOC-T)  and  ensure 
created  test  environment  is  operationally  relevant 

•  Joint  Mission  Environment  Design  Working  Group 
(JDWG)  -  Develop  the  JBD2  Logical  Design,  Physical 
Design,  and  System  Description  Document  (SDD) 

•  Test  Integration  Working  Group  (TIWG)  -  Responsible 
for  network  infrastructure,  implementation  of  Physical 
Design,  integration  of  systems,  and  execution  of  test 

•  Test  Design  and  Analysis  Working  Group  (TDAWG)  - 

Develop  overall  test  design,  ensure  required  data  is 
collected,  and  perform  analysis  of  test  results 

This  paper  will  focus  on  the  roles  and  responsibilities  of  the 
Test  Integration  Working  Group  and  will  present  lessons 
learned  from  the  TIWG  perspective.  The  overview  in  this 
paper  is  intended  to  provide  enough  information  about  JBD2 
to  put  the  TIWG  lessons  learned  in  context.  However,  it  is  not 
a  complete  summary  of  the  JBD2  test  event. 
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1)  TIWG  Overview 

The  TWIG  was  a  small  technically  focused  group  responsible 
for  test  integration,  test  execution,  network  infrastructure,  and 
network  security.  The  TIWG  selected  and  distributed  all  test 
execution  tools  necessary  for  the  conduct  of  the  JBD2  test. 
Integration  of  the  JME  was  conducted  through  integration 
spiral  events  coordinated  by  the  TIWG  through  the  Integration 
Spiral  Plan.  Test  integration  activities  were  done  in 
accordance  with  the  Physical  Design  Documents  and  the 
Detailed  Test  Plan.  All  details  of  the  test  integration  process 
were  documented  in  spiral  reports  completed  at  the  end  of 
each  spiral.  During  the  dry  run  week  and  test  week,  situation 
reports  were  generated  and  distributed  on  a  daily  basis.  The 
TIWG  process  follows  the  hierarchal  design  presented  in 
Fig.  1. 
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Figure  1 .  Integration  Waterfall  Diagram 


concept  section  concludes  with  the  live,  virtual,  constructive 
distributed  environment  LVC-DE  design  and  a  description  of 
the  test  venue. 

The  overall  objective  of  the  JBD2  Test  Event  is  to  assess  the 
degree  the  blue  force  can  successfully  conduct  identified 
missions  in  the  context  of  controlled  changes  to  specific 
materiel  and  non-materiel  factors,  holding  other  factors 
constant.  The  two  battlespace  management  systems  are  the 
Theater  Battle  Management  Core  Systems  (TBMCS)  and 
Tactical  Airspace  Integration  System  (TAIS). 

TBMCS  provides  the  combat  air  forces  and  the 
Joint/combined  forces  with  an  automated  and  integrated 
capability  to  plan  and  execute  the  air  battle  plan  for  operations 
and  intelligence  personnel  at  the  combined  air  operations 
center  and  individual  unit  levels.  It  provides  the  air 
commander  with  the  means  to  plan,  direct,  and  control  all 
theater  air  operations  in  support  of  command  objectives.  It 
also  coordinates  with  engaged  ground  and  maritime  elements. 

TAIS  is  the  Army’s  materiel  solution  for  the  integration  and 
synchronization  of  Airspace  Command  and  Control  (AC2)  and 
en  route  Air  Traffic  Services  (ATS)  within  the  Army  Battle 
Command  System  (ABCS). 

A.  Test  Design 

This  SoS  is  a  C2  capability  which  will  analyze  two  factors. 
The  first  factor  will  be  the  battlespace  management  capability 
(current  -  TBMCS/TAIS  9.3  versus  future  -  TBMCS/TAIS 
10.0).  The  second  factor  is  the  C2  processes  utilized  by  the 
Joint  forces  (current  procedural-based  processes  versus  future 
“expedited”  processes).  Each  of  these  factors  is  comprised  of 
two  levels  (current  level  and  future  level). 


B.  SYSTEM  DESCRIPTION 

To  support  the  JBD2  test  event  a  notional  SoS  was  assembled 
to  provide  the  capability  to  deconflict  near-real-time  tactical 
changes  during  the  full  range  of  military  operations. 


Battle  Space  Management 

The  materiel  test  factor  attempts  to  discern  if  a  difference 
exists  between  the  current  and  future  SoS  being 
implemented: 


Airspace  control  procedures  provide  maximum  flexibility 
through  an  effective  mix  of  positive  and  procedural  control 
measures.  The  control  structure  encourages  close  coordination 
between  components  to  allow  rapid  concentration  of  combat 
power.  The  methods  of  airspace  control  vary  throughout  the 
range  of  military  operations.  They  range  from  positive  control 
of  all  air  assets  in  an  airspace  control  area  to  procedural 
control  of  all  such  assets,  or  any  effective  combination  of  the 
two.  Airspace  control  procedures  and  systems  need  to 
accommodate  these  methods  based  on  component.  Joint  and 
national  capabilities,  and  requirements. 

II.  Test  Concept 

The  following  sections  describe  the  overall  goal  and  test 
objective  of  the  JBD2  test  event.  The  test  design  provides  the 
test  factors  and  their  respective  levels  as  well  as  the  potential 
test  execution  matrices.  Additionally,  an  overview  of  the  test 
scenario  and  the  specific  mission  tasks  of  interest  which 
provide  operational  context  for  the  test  are  included.  The  test 


•  Current:  implementation  with  TBMCS  and  TAIS  9.3 

•  Future:  implementation  with  TBMCS  and  TAIS  1 0.0 

Timeliness  of  C2  processes 

The  non-materiel  factor  focuses  on  differences  in  the  C2 
processes  as  either  the  current  procedural-based  process 
versus  a  future  process. 

•  Current:  procedural  based  process 

•  Future:  “expedited”  based  process 

Factors  and  Levels  Combinations 

Each  of  the  four  combinations  of  factor  levels  is  a  trial.  Each 
conduct  of  a  trial  is  a  run.  Table  I  shows  the  factors,  levels, 
and  associated  notations. 


TABLE  1.  EXPERIMENTAL  FACTORS,  LEVELS,  AND  NOTATION 


Factor  1 

Factor  2 

Trial 

Battlespace 

Timeliness  of  C2 

Run  1 

Run  2 

Management 

Processes 

Current 

Current 

1 

TAIS  9.3 

proeedural 

[C.C], 

[C,C]2 

Current 

Future 

2 

TAIS  9.3 

expedited 

[C.F], 

[C,F]2 

Future 

Current 

3 

TAIS  10.0 

proeedural 

tF,C]i 

tF,C]2 

Future 

Future 

4 

TAIS  10.0 

expedited 

[F.F], 

[F,F]2 

B.  Operational  Context 

The  JBD2  test  event  is  focused  on  six  key  mission  tasks  as 
listed  in  Table  II  and  is  further  categorized  by  the  mission 
type.  During  the  test  event,  these  key  mission  tasks  occur 
simultaneously  and/or  sequentially  as  the  scenario  executes 
according  to  the  master  scenario  event  list  (MESL). 


TABLE  IT  KEY  MISSION  TASKS 


Type 

Mission  Task 

US  Army  (USA)  Maneuver  (MVR)  Observer  to  US  Marine 
Corps  (USMC)  High  Mobility  Artillery  Roeket  System 
(HIMARS) 

Joint  Fires 
(JFIRES) 

US  Marine  Corps  (USMC)  Maneuver  (MVR)  Observer  to  US 
Army  (USA)  Non-Line  of  Sight  -  Launeh  System  (NLOS-LS) 

US  Air  Foree  (USAF)  Unmanned  Aerial  Sensor  (UAS)  to  US 
Army  (USA)  Non-Line  of  Sight  -  Launeh  System  (NLOS-LS) 

US  Air  Foree  (USAF)  Unmanned  Aerial  Sensor  (UAS)/  US 
Army  (USA)  Forward  Support  Element  (FSE)  to  USAF 
Weaponized  UAS  (w/SWARM) 

Joint 

Close  Air 

Support 

(JCAS) 

US  Air  Foree  (USAF)  Joint  Terminal  Attaek  Controller 
(JTAC)  to  US  Air  Foree  (USAF)  Fixed  Wing 

us  Marine  Corps  (USMC)  Joint  Terminal  Attack  Controller 
(JTAC)  to  Fixed  Wing  (US  Air  Force  (USAF)  /  US  Navy 
(USN)) 

Fig.  2  shows  the  JBD2  Operational  View  1  (OV  -1)  with  each 
of  the  six  Key  Mission  Tasks  represented  with  the  major 
sensors,  weapons,  and  C2  systems. 


Joint  Fires  /  Joint  CAS 
Air  Maneuver/Operations 
Combat  Identification 
f?  Battlespace  Deconfliction  C2 


Figure  2.  JBD2  Operational  View  1  (OV-1) 


C.  Live  Virtual  Constructive  -  Distributed  Environment 
(L  VC-DE)  Design 

The  JBD2  test  event  was  executed  in  a  Joint  environment 
across  1 6  different  test  sites  with  live,  virtual,  and  constructive 
elements  connected  through  the  JMETC  VPN.  All  Wide  Area 
Network  (WAN)  simulation  traffic  was  exchanged  using  the 
Testing  and  Training  Enabling  Architecture  (TEN A).  Each 
system  involved  in  the  test  was  time  synchronized  to  local 
NTP  servers  or  GPS.  The  JBD2  test  event  terrain  is  a  4  X  3 
degree  region  in  the  area  of  Fort  Bliss,  Texas,  and  White 
Sands  Missile  Range  (WSMR). 

The  tactical  communication  standards  used  in  the  L VC-DE 
included  Tactical  Digital  Information  Link  (TADIL-J), 
Variable  Message  Format  (VMF),  and  the  United  States 
Message  Text  Format  (USMTF).  The  JBD2  infrastructure 
supported  tactical  communications  across  the  test  WAN. 

The  simulation  architecture  for  JBD2  was  driven  by  the 
simulations  selected  to  participate,  rather  than  the  architecture 
driving  which  systems  were  selected.  This  approach  resulted 
in  a  mixed  simulation  architecture  using  multiple  simulation 
protocols  including  Distributed  Interactive  Simulation  (DIS), 
the  High  Level  Architecture  (HLA),  and  the  Test  and  Training 
Enabling  Network  Architecture  (TEN A). 


A  goal  from  the  start  of  JBD2  test  planning  was  to  take  full 
advantage  of  the  JMETC  infrastructure  and  tools  in  the  design 
of  the  JBD2  simulation  architecture.  JMETC  has  selected 
TENA  as  the  common  simulation  middleware  for  achieving 
Joint  interoperability  between  DoD  ranges.  TENA  was 
selected  to  be  the  only  simulation  protocol  used  across  the 
WAN.  JMETC  gateways  were  used  at  each  site  that  required 
DIS  or  HLA  traffic  on  their  LANs.  Each  lab  using  HLA 
simulations  ran  an  isolated  HLA  Federation  and  Run-time 
Infrastructure  Execution  (RTIExec). 

D.  Test  Venue 

The  venue  for  JBD2  test  event  is  a  distributed  test  event 
linking  test  facilities  and  sites  in  order  to  compose  a  LVC-DE. 
Each  capability  provider  is  linked  using  the  JMETC 
infrastructure  in  order  to  compose  the  JME  required  for  the 
test  event.  Fig.  3  provides  an  illustration  of  the  test 
participants’  organizations  with  their  supporting  roles  and 
systems. 

The  JMETC  Virtual  Private  Network  (VPN)  was  established 
on  the  High  Performance  Computing  Modernization  Program 
Office  -  Secure  Defense  Research  and  Engineering  Network 
(HPCMPO-SDREN).  The  JMETC  VPN  enabled  use  of 
JMETC  System  Control  (SYSCON)  tools  for  network  quality 
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Figure  3.  JBD2  System  View  1  (SV-1) 


control  and  supports  effective  use  of  other  JMETC  products. 
JMETC  used  the  Network  Aggregation  Router  to  bridge  to 
other  secure  networks,  such  as  the  Air  Force  Integrated 
Collaborative  Environment  (AF-ICE)  and  the  Joint  Training 
and  Experimentation  Network  (JTEN).  JDB2  required  six 
new  VPN  sites  with  two  of  the  sites  connected  via  the 
Network  Aggregation  router. 

•  JMETC  VPN 

-  Redstone,  RTTC  DTTC 

-  Redstone,  RTTC  OMAN 

-  Pax  River,  ACETEF 

-  White  Sands,  IRCC 

-  Eglin  AFB,  GWEF 

-  Eglin  AFB,  46TS  C2TF 

-  JFCOM,  JSIC 

-  Aberdeen,  ACCN 

-  Ft.  Hood,  OTC-TTEC 

-  Charleston  SSC,  Bldg.  3147 

-  Charleston  SSC,  Bldg.  3440 

-  Ft.  Fewis,  EPG 

•  JTEN  Enclave 

-  JFCOM,  Test  Bay  8 

•  AF-ICE  Enclave 

-  Wright  Patterson  AFB,  SIMAF 

-  Fangley  AFB,  GCIC 

-  Pentagon,  WARCAP 


III.  Fessons  learned 

The  lessons  learned  portion  of  this  paper  is  divided  into 
sections  that  correspond  to  different  aspects  of  the  integration 
and  test  process:  Pre-Test/Integration,  Test  Sites  and 

Network,  Test  Execution,  and  General.  Recommendations  for 
future  events  and  investments  are  also  included  with  the 
description  of  the  lessons  learned. 

A.  Pre-test/Integration 

During  the  pre-test/integration  phase  of  JBD2,  several  lessons 
were  learned.  First,  it  is  recommended  that  site 
surveys/certification  be  conducted  prior  to  selecting  which 
sites  will  participate  in  the  test.  The  site  surveys  should 
determine  if  the  site: 

•  can  support  computers  systems  running  both  Finux  and 
Windows  operating  systems. 

•  is  approved  for  classified  open  storage.  This  is  critical 
in  a  classified  distributed  test. 

•  will  have  enough  personnel  available  during  spiral 
integration  and  test  runs  (all  sites  should  have  a 
minimum  of  2  people  in  the  lab). 

•  has  back-up  personnel  available  for  key  positions  being 
hosted  at  that  site. 

•  has  enough  commercial  and  VoIP  lines  available  in  the 
lab.  A  site  should  have  at  least  2  VoIP  phones  and  2 


commercial  lines.  These  phones  should  be  located  next 
to  key  systems. 

•  has  enough  computers  to  run  all  required  applications. 
Running  multiple  network-intensive  or  graphics 
intensive  applications  on  a  single  computer  should  be 
avoided. 

If  a  site  is  selected  to  participate  in  the  event,  performing  on¬ 
site  integration  of  required  tools  (protocol  gateways,  test  tools, 
etc.)  should  be  considered. 

Additionally,  during  the  pre-test  phase,  a  complete  list  of 
required  IP  addresses,  ports  and  protocols  should  be  defined 
early  in  the  process.  During  JBD2,  the  majority  of  network 
complaints  and  issues  were  related  to  undocumented  ports  and 
IP  addresses.  Not  having  the  IP  addresses,  ports,  and 
protocols  defined  by  all  sites  early  had  a  definite  impact  on  the 
integration  and  dry  run  spirals. 

Several  applications  were  integrated  very  late  in  the  JBD2 
schedule  with  minimal  integration  testing  and  documentation. 
It  is  essential  for  new  applications  (or  for  significant  changes 
to  existing  applications)  be  documented,  tested,  and  validated 
before  the  integration  spirals  being.  The  lead  range  should 
support  one-on-one  testing  as  needed  during  early  integration 
to  help  with  application  testing.  All  applications  playing  in  a 
distributed  event  should  have  design  documentation  including 
enumeration  data  and  this  documentation  should  be  provided 
to  event  organizers  early  in  the  process.  It  is  important  to  note 
that  development  and  testing  of  the  protocol  gateways  cannot 
be  completed  until  all  application  and  enumeration  data  is 
submitted. 

Multiple  integration  spirals  are  required  to  integrate  a  large 
multi-site  distributed  test  event.  The  number  and  extent  of  the 
spirals  is  greatly  dependant  on  the  level  of  distributed  test 
maturity  at  the  participating  sites,  required  interoperability 
between  the  sites,  the  number  of  new  tactical  applications,  and 
the  number  of  new  test  tools  being  integrated.  Plans  for  these 
integration  spirals  should  contain  a  detailed  schedule  of 
activities.  However,  sites  must  be  flexible  as  planned 
activities  may  be  shorter  or  longer  than  expected.  Once  the 
initial  site  integrations  are  complete,  spirals  in  which  all  sites 
cannot  fully  participate  have  reduced  value  since  full 
integration  testing  cannot  be  accomplished  until  all  sites  are 
online.  Development  of  a  “Test  Harness”  class  of  tool  is 
needed  to  aid  sites  in  certifying  their  ability  to  exchange  data 
properly  before  entering  integration. 

During  JBD2  multiple  integration  activities  were  performed 
simultaneously  during  a  spiral.  For  concurrent  activities  to  be 
effective,  sites  need  several  personnel  and  communication 
lines  available.  Each  simultaneous  activity  needs  its  own  lead. 
Additionally,  an  end-of-the-day  “hot  wash”  should  occur 
where  all  sites  discuss  their  accomplishments. 

It  is  also  good  practice  to  have  a  backup  plan  for  all  critical 
applications  and  sites.  Although,  this  will  increase  integration 
activities  and  cost,  it  is  essential  during  large  distributed  tests. 


During  the  LVC-DE  integration  spirals,  the  inclusion  of 
analytical  requirements  as  part  of  the  objective  facilitated  early 
utilization  of  the  data  collection  process  and  identified  data 
collection  issues.  The  early  involvement  improves  process 
familiarization,  increases  the  speed  of  the  data  reduction,  and 
directly  supports  the  Verification  &  Validation  (V&V) 
process. 

B.  Test  Sites  and  Network 

All  test  sites  should  have  organic/local  test  network  support. 
This  support  should  allow  each  lab/range  to  quickly  react  to 
problems.  If  the  site  does  not  have  network  support  available 
in  the  same  building,  the  site  must  work  with  their  network 
support  organization  to  have  someone  on  call  and  readily 
available.  Several  times  during  JBD2,  the  integration  was 
delayed  while  awaiting  network  personnel  to  respond  to  a  call. 
Often  it  was  a  simple  operation  that  needed  to  be  completed, 
but  it  would  take  several  hours  while  awaiting  network 
personnel  to  respond  (often  having  to  drive  across  the  base  to 
do  so). 

To  the  extent  possible,  participating  sites  should  not  have 
firewalls.  Issues  with  firewalls  consumed  a  significant  amount 
of  integration  time.  If  firewalls  cannot  be  avoided  at  a  site, 
modifications  to  the  firewall/access  control  list  should  be 
made  by  classes  or  ranges  of  IPs  addresses,  not  by  individual 
addresses  or  ports. 

In  a  multi-architecture  test  environment  gateways  are  a  critical 
element.  When  using  new  gateway  building  tools,  onsite 
support  from  the  tool  developer  may  be  needed.  Many 
gateway  builder  bugs  were  found  during  the  JBD2  integration 
spirals.  Unfortunately,  due  to  the  late  integration  of  several 
applications  and  hence  the  late  completion  of  the  gateways, 
load  testing  (throughput,  latency)  was  not  completed  on  the 
gateways  before  the  JBD2  event  runs  for  record.  It  is 
recommended  that  remote  control  capabilities  be  added  to  the 
gateways  for  future  events.  This  capability  would  allow 
personnel  to  more  easily  troubleshoot  gateway  problems  at 
remote  sites.  Additionally,  the  network  characterization  and 
test  tools  should  have  the  capability  of  remote  operation.  A 
quick  look  network  performance/health  tool  would  be  a 
valuable  asset  to  have  during  distributed  tests. 

A  real-time  TENA  capable  Cross  Domain  Solution  (CDS)  is 
needed  between  classified  and  unclassified  test  networks  to 
allow  visualizing  tests  and  performing  selected  data  analysis  in 
an  unclassified  environment.  Additionally,  a  test  network 
design  approach  that  will  handle  the  tactical  IP  address  space 
(non-routable  addresses)  must  be  developed. 

The  Joint  test  community  must  work  with  the  various  DoD 
Information  Assurance  (lA)  departments  to  develop  consistent 
service-level  lA  requirements,  i.e.,  the  lA  requirements  should 
be  the  same  from  service -to- service.  It  is  difficult  to  share 
tools  and  applications  if  the  lA  and  software  certification 
requirements  and  language  are  not  consistent.  Additionally,  a 
method  for  Security  Classification  Guide  (SCG)  development 
to  support  Joint  distributed  testing  needs  to  be  determined. 


C.  Test  Execution 

During  the  test  event  there  were  several  tools  that  proved  to  be 
useful.  Having  a  classified  wiki  (wiki  is  a  type  of  website  that 
allows  users  to  add,  remove,  and  edit  the  content)  was  very 
important  for  information  sharing  after  the  concept  was 
embraced  by  all  sites.  The  unclassified  wiki  was  used  more  in 
the  test  planning  and  initial  integration  activities.  White  Sands 
Missile  Range  (WSMR)  provided  a  persistent  commercial 
conference  phone  line  that  was  available  for  the  whole  event. 
The  Distributed  Capabilities  Integration  Toolbox  (DCIT)  was 
very  useful  for  integrating  such  a  large  distributed  event  by 
providing  a  single  place  for  sites  to  check-off  important  events 
as  they  were  accomplished.  The  Digital  Collection,  Analysis, 
and  Review  System  (DCARS)  was  useful  for  distributed  data 
collection.  The  DCARS  LAN  Data  Collector  (LDC)  data 
stream  to  the  DCARS  Data  Processing  Units  (DPUs)  was  the 
largest  source  of  data  on  the  network.  This  was  anticipated  for 
JBD2  based  on  the  data  collection  design  but  future  test  event 
designers  should  pay  careful  attention  to  the  data 
collection/network  architecture. 

One  test  resource  tool  that  was  not  made  available  during 
JBD2  was  a  test  conductor  chat  capability.  This  would  have 
been  a  very  useful  tool  to  augment  other  forms  of 
communication  (Voice,  VTC,  wiki,  etc.).  Another  useful 
communication  tool  was  the  classified  Video  Teleconference 
(VTC)  capability  that  existed  between  two  of  the  test  labs. 
Adding  this  capability  to  all  sites  would  aid  in  communication 
if  the  network  bandwidth  is  available.  During  test  execution 
(spirals,  dry  runs,  runs  for  record),  it  is  recommended  that  one 
person  be  assigned  to  monitor  the  test  control  communication 
line  at  all  times. 

Before  the  start  of  every  test,  each  site  was  required  to  follow 
a  specific  start-up  sequence  to  ensure  certain  essential 
activities  were  performed.  The  TestTalk  tool  was  used  to 
display  the  official  “run  clock”  for  all  sites  and  to  display  the 
site  specific  Time  Ordered  Event  List  (TOLL).  This  TOLL 
was  what  each  site  followed  during  the  start-up  sequence. 
This  approach  proved  very  beneficial  in  starting  the 
complicated  JME  in  a  structured,  repeatable  manner. 

At  the  end  of  the  day,  daily  situation  reports  (SITREPS)  were 
compiled  and  distributed  to  a  large  audience.  This  proved  to 
be  a  very  valuable  way  to  distribute  the  daily  progress  of 
JBD2.  However,  there  needs  to  be  more  granularity  in  the 
SITREPS.  It  is  recommended  that  one  be  generated  for  the 
operators  of  the  test  and  one  generated  for  management  (more 
high-level,  less  technical  details.).  Also,  to  expedite  the 
compilation  of  the  SITREPs,  a  procedure  for  data  transfer 
across  security  domains  is  essential  at  the  lead  and/or  data 
archival  site. 

D.  General 

Complicated  operational  scenarios  with  multifaceted  tactical 
communications,  such  as  the  one  used  in  JBD2,  require  more 
spiral  integration  events.  Several  spirals  need  to  be  devoted  to 
working  operational  issues,  and  these  spirals  cannot  be 


performed  until  the  technical  integration  is  complete.  More 
operator  training  is  necessary  if  technical  personnel  are 
required  to  play  an  operational  role  or  use  operational 
equipment.  It  is  recommended  that  at  least  one  full  spiral  be 
devoted  to  this  training. 

Test  sites  need  to  consider  and  plan  resources,  such  as  space, 
hardware  (computers,  VoIP,  radio),  and  software  (web-based, 
data  collection/reduction)  to  support  observers  and  analyst 
efforts  to  evaluate  the  system  under  test.  The  use  of  chat 
rooms  in  battle  command  centers  has  become  common  place, 
but  the  ability  to  capture  and  analyze  the  data  is  lacking. 
Recommend  the  T&E  community  identify  methods  and 
process  as  well  as  applications  to  capture  and  analyze  the 
information  communicated  by  these  technologies. 

As  technical  maturity  evolves,  test  programs  utilizing  tactical 
command  and  control  infrastructure  should  use  tactical 
network/communication  simulations  versus  the  “perfect” 
connections  utilized  in  JBD2  test.  The  addition  of  the  tactical 
network  simulation  is  needed  to  provide  realistic  test  scenarios 
for  the  network  centric  environment. 

More  problem  correction  time  is  needed  between  the 
integration  spirals  until  the  architectures,  implementations,  test 
sites,  and  processes  mature.  Test  sites  must  develop  a  respect 
for  the  rigor  of  testing  in  a  LVC  distributed  environment. 
Persistent  networks,  test  infrastructure,  and  continual 
employee  training  are  needed  as  well  as  periodic  integration 
activities  with  other  organizations  to  make  LVC  distributed 
testing  executable  in  a  realistic  timeframe  and  budget. 

IV.  Conclusions 

The  JBD2  test  event  was  a  major  leap  forward  in  the  test 
planning,  test  conduct,  and  evaluation  of  systems  in  a  Joint 
distributed  test  environment.  In  all  Joint  testing,  cultural 
challenges  exist  and  JBD2  made  significant  progress  in 
breaking  down  the  barriers.  JBD2  was  a  real  test  program 
with  test  factors  and  levels  and  a  true  factorial  design. 

With  a  persistent  network  and  common  test  tools  and 
standards,  the  vision  of  a  Joint  Mission  Environment  that 
enables  system  testing  of  Joint  requirements  is  an  attainable 
goal.  Also  key  to  a  successful  JME  are  the  perishable 
commodities  at  each  of  the  test  facilities.  Assets  such  as 
personnel,  training,  system  configurations,  tools, 
middleware/protocol  versions,  etc.  must  be  maintained. 

JBD2  was  a  success,  but  there  are  a  number  of  issues  and 
recommendations  that  need  to  be  addressed  by  the  T&E 
community.  Below  is  an  executive  summary  list  of  the  major 
lessons  learned/recommendations  generated  by  the  JBD2 
TIWG. 

•  Site  surveys/certification  should  be  conducted  prior  to 
finalizing  site  lists 

•  Adequate  voice  communications  are  needed  in  labs  (2-3 
VoIP  &  2  commercial  lines) 


•  All  sites  should  be  approved  for  open  storage  when 
conducting  a  classified  test 

•  Complete  list  of  required  IP  addresses.  Ports  and 
Protocols  should  be  defined  early  in  process 

•  Sites  should  have  organic/local  network  support 

•  Test  sites  should  not  have  network  firewalls  if  at  all 
possible 

•  Classified  Wiki  was  very  valuable  during  JBD2  event 
integration  and  execution 

•  Persistent  networks,  test  infrastructure,  and  continual 
employee  training  are  needed 

•  Periodic  integration  activities  with  participating 
organizations  is  needed  to  make  LVC  distributed  testing 
executable  in  a  realistic  timeframe  and  budget 
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JBD2  JTEM  08  Test  Event  Background 

and  Rationale 


Future  Combat  Systems  (FCS) 

•  Oct  2002:  FCS  Test  (IS&T/CTO) 
recognized  that  FCS  presents  new  testing 
challenges  that  require  new  test  capabilities 

-  Systems  of  Systems  on  a  grand  scale 

-  Move,  shoot ,  communicate  - 
simultaneously 

-  Seamless  integration  with  Joint 
elements  for  Network-centric 
operations 

•  2003  and  2006  TEMP  codified  the  need  for 
investments  to  provide  needed  test 
capability 

-  Instrumentation 

-  Modeling  and  Simulation 

-  Networking 

•  Multiple  investments  initiated  to  provide 
needed  stimulation,  data  collection  and 
analysis  requirements 

•  Many  of  these  investments  are  maturing 
and  need  to  be  tested 


DoD  Joint  Test  Capability  Roadmap 

•  Mar  2003:  DoD  recognized  that  it’s  abiiity 
to  impiement  seamiess  Joint  operations 
is  insufficient 

•  2004  DoD  Strategic  Pianning  Guidance 
demanded  creation  of  Joint  environment 
testing  capabiiity 

•  2005:  OSD  Initiated  Joint  Test  and 
Evaiuation  Methodoiogy  (JTEM)  project 
and  Joint  Mission  Environment  Test 
Capabiiity  (JMETC)  deveiopment  effort 

•  JTEM  focuses  on  the  methods  and 
processes  associated  with  testing  in  a 
Joint  environment 

•  JMETC  focuses  on  the  infrastructure 
needed  to  faciiitate  the  JTEM  methods 
and  processes 

•  JTEM  and  JMETC  have  created  test 
methods  and  infrastructure  that  need  to 
be  tested 


2008  Event  Objective:  Assess  suitability  of  JTEM  CTM  for  testing  FCS  test 
technologies  using  JMETC  infrastructure  in  a  relevant  Operational  context 
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JBD2  JTEM  08  Test  Event  Goals 


•  JTEM  (Joint  Test  and  Evaluation  Methodology):  Assess  effectiveness  and  suitability  of 
JTEM  Capability  Test  Methodology  (CTM)  processes 


•  JMETC  (Joint  Mission  Environment  Test  Capability):  Characterize  the  network 
infrastructure  and  mature  the  baseline  capability  to  support  SoS  level  testing  across  the 
Services 


•  PCS  (Future  Combat  Systems):  Establish  a  rigorous  test  context  to  examine  PCS  test 
technology  requirements  needed  for  testing  in  a  Joint  environment  in  support  of  PCS 
Milestone  C  test  activities 

-  Application  of  JTEM  methods  and  processes;  use  of  JMETC  infrastructure 

-  Perform  test  capability  assessment  within  JBD2  context 

-  Utilize  PCS  CTO  Test  Technology  investments  to  baseline  this  context 

•  Common  Control  Nodes  (CCN) 

•  Test  Center  Assets 

•  Test  Tools  and  Instrumentation 

•  PCS:  Incremental  build  up  of  critical  PCS  test  technology  areas  in  a  Joint  Operational  context 

-  Network  testing  support  technologies  (e.g.  Representation  of  interactions  between  Joint 
Platforms) 

-  Distributed  Test  Infrastructure  Technologies  (e.g.  JMETC  and  InterTEC  Technologies) 

-  VV&  Aofthe  LVC-DE 


Support  of  Joint  Initiatives  While  Addressing  PCS  Test  Technology  Objectives 
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*■  vJTAC 

-  JRE-LINK  16/11 
-ASTi 

f  DREAMS 
GWEF,  Eglin  AFB 

-  SWARM 
-ASTi 


sCIC,  Langley  AFB 

GCCS-J 

-TBMCS 

-ASTi _ 

Iecs^pawa^^" 

GCCS-M 
TBMCS 
CEC  RTFS 
-ASTi 

JSMC,  SPAWAR  -  C 

-  GCCS-J 
TBMCS 

I  AFATDS 

ASTi _ 
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Key  Mission  Tasks 


Type 

Mission  Task 

US  Army  (USA)  Maneuver  (MVR)  Observer  to  US  Marine 
Corps  (USMC)  Jfigh  Mobility  Artillery  Rocket  System 
(HIMARS) 

Joint  Fires 
(JFIRES) 

US  Marine  Corps  (USMC)  Maneuver  (MVR)  Observer  to  US 
Army  (USA)  Non-Line  of  Sight  -  Launch  System  (NLOS-LS) 

US  Air  Force  (USAF)  Unmanned  Aerial  Sensor  (UAS)  to  US 
Army  (USA)  Non-Line  of  Sight  -  Launch  System  (NLOS-LS) 

US  Air  Force  (USAF)  Unmanned  Aerial  Sensor  (UAS)/  US 
Army  (USA)  Forward  Support  Element  (FSE)  to  USAF 
Weaponized  UAS  (w/SWARM) 

Joint 

Close  Air 

Support 

(JCAS) 

US  Air  Force  (USAF)  Joint  Terminal  Attack  Controller 
(JTAC)  to  US  Air  Force  (USAF)  Fixed  Wing 

US  Marine  Corps  (USMC)  Joint  Terminal  Attack  Controller 
(JTAC)  to  Fixed  Wing  (US  Air  Force  (USAF)  /  US  Navy 
(USN)) 

Joint  Battlespace  Dynamic  Deconf action: 
Standard  Mission  Task  Fiow 
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Test  Factors  and  Levels 


Trial 

Factor  1 

Battlespace 

Management 

Factor  2 

Timeliness  of  C2 
Processes 

Runl 

Run  2 

1 

Current 

TAIS  9.3 

Current 

procedural 

[C,C]i 

[C,C]2 

2 

Current 

TAIS  9.3 

Future 

expedited 

[C,F], 

[C,F]2 

3 

Future 

TAIS  10.0 

Current 

procedural 

[F.C], 

[F,C]2 

4 

Future 

TAIS  10.0 

Future 

expedited 

[F,F], 

[F,F]2 
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JBD2  Site  Connectivity 


JBD2  JTEM  08  Integration  Plan 


Support  of  Analysis  Activities 


Exe  of  Service  Level  “On-Line”  Initiatives 


Test  Infrastructure  Integration 


Network  Connectivity 
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JBD2  JTEM  08 


Lessons  Learned 
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Site  Survey 

It  is  recommended  that  site  surveys/certification  be 

conducted  prior  to  selecting  which  sites  will  participate  in 

the  test.  The  site  surveys  should  determine  if  the  site: 

•  can  support  computers  systems  running  both  Linux  and  Windows  OS 

•  is  approved  for  classified  open  storage. 

•  will  have  adequate  personnel  available  during  spiral  integration  and 
test  runs 

•  has  back-up  personnel  available  for  key  positions  being  hosted  at  that 
site. 

•  has  enough  commercial  and  VoIP  lines  available  in  the  lab  (2  VoIP 
and  2  commercial  min.).  These  phones  should  be  located  next  to  key 
systems. 

•  has  enough  computers  to  run  all  required  applications.  Running 
multiple  network-intensive  or  graphics  intensive  applications  on  a 
single  computer  should  be  avoided. 
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Networking 


•  IP  ADDRESSES,  PORT  &  PROTOCOLS!  -  few 
understood  the  impact  to  the  integration  cycles  and  dry 
runs  caused  by  IP  addresses  and  ports  numbers  not 
being  integrated  into  the  firewalls.  The  majority  of  all 
network  complaints  and  issues  were  related  to 
undocumented  ports  and  IP  addresses. 

•  It  is  preferred  that  each  lab  participating  in  the  test  event 
have  organic  network  support.  This  allows  each  lab  to 
quickly  react  to  problems. 

-  If  a  lab  does  not  have  organic  network  support,  they  should  work 
with  their  network  support  organization  to  have  someone  on  call. 

•  It  is  preferable  that  each  lab  have  access  to  their 
TACLANE  to  perform  routine  actions  such  as  deleting 
calls.  Where  this  is  not  possible,  personnel  with  access 
to  the  TACLANE  need  to  be  on  stand-by. 
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Networking  (Cont) 


•  To  the  extent  possible  participating  sites  should  not  have 
firewalls.  Issues  with  firewalls  consumed  a  large  about  of 
integration  time.  If  firewalls  can  not  be  avoided  at  a  site, 
modifications  to  the  access  control  list  should  be  made  by 
classes  or  ranges  of  IPs,  not  individual  addresses  or  ports 


•  Network  configuration  management  and  documentation 
must  improve  to  have  a  repeatable  test  process 


•  Micro  TACLANE  issue  uncovered  in  JBD2  should  be 
investigated  until  root  cause  is  determined 
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Comm  unica  tions 


•  WSMR  persistent  phone  line  was  wonderful!  Available  for  whole 
event. 

•  All  sites  need  at  least  2  and  preferably  3  VoIP  phones 

-  1  for  test  control  coordination 

-  1  for  operation  coordination  and  operational  communication  backup 

-  1  for  back-line  trouble  shooting 

•  All  sites  need  at  least  2  commercial  lines 

-  1  for  hotwash  &  VoIP  test  control  backup 

-  1  for  back-line  communications 

•  Ideally  phones  should  be  located  next  to  key  systems.  Some  sites 
reported  that  the  phones  were  across  the  room  from  key  systems. 

•  Sites  should  have  an  amplified  test  control  line  that  can  be  heard 
throughout  the  room.  Speaker  phones  do  not  work  well  for  this. 
Some  sites  prohibit  such  configurations. 

•  All  personnel  talking  on  the  test  control  line  should  be  on  headsets 
not  speaker  phones. 
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Personnel 


•  Sites  should  have  backup  staff  for  key  positions.  In  a  test 
event  lasting  several  months,  staff  will  be  sick  or  have 
other  obligations. 

•  During  test  execution  (spirals,  dry  runs,  run  for  record), 
one  person  should  be  assigned  to  monitor  the  test  control 
line. 

•  More  training  for  test  resources  and  operational  systems 
is  required. 

•  Sites  must  develop  a  respect  for  the  rigor  of  “testing”  in 
LVC  distributed  environment. 
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Applications 


New  applications  or  significant  changes  to 
applications  must  be  documented,  tested,  and 
validated  before  the  integration  spirals  begin. 
Some  testing  can  only  occur  during  the 
spirals,  but  this  should  be  minimized. 


DCIT  is  very  useful  for  integrating  large 
distributed  events. 


TENA  to  DIS/MATREX  Gateway  Comments 

-  Can  not  complete  GW  development  work  and 
testing  until  application  data  is  submitted. 
Several  applications  came  in  very  late. 

-  JMETC  provided  good  support  from  but  GW 
builder  developer  on  site  support  is  needed. 
Many  GW  builder  code  bugs  were  found  during 
the  Integration  Spirals. 

-  Remote  control  capabilities  should  be  added  to 
the  GW  software. 


DCIT:  Connectivity  Matrix  Utility  (CMU) 


•  a004AKTtCC<iip<CTOOft-DCTr^c»>iiii03d 
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Applications  (Cont) 


•  DCARS 

-  Add  capability  to  collect  native  AFATDS  traffic 

-  Add  capability  to  collect  ASTi  DIS  traffic 

-  The  startup  sequence  was  the  same  for  every  run  -  need  automation  script 

-  Determine  path  ahead  to  solve  issue  with  HLA  collector  feeding  RPWS  (idea  -  if 
buffer  fills,  throw  away  out  of  date  data) 

-  LAN  Data  Collector  (LDC)  stream  to  DPUs  was  the  largest  source  of  data  on 
network 

-  DCARS  documentation  should  be  revised  to  remind  installers  that  DISA  Gold  should 
be  run  after  installation  of  all  software  on  the  platform  to  ensure  that  the  required 
ports  are  not  blocked. 


•  Need  to  incorporate  a  test  conductor  chat  capability 

•  We  need  to  understand  OneSAF  lethality  and  weapon  modeling  better 


Test  Talk 

-  An  out  of  date  version  of  Test  Talk  was  inadvertently  use  in  the  test  (Lead  Range  POC  error) 
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Miscellaneous 


•  Having  a  classified  Wiki  was  very  important  for 
information  sharing  after  the  concept  was  embraced  by  all 
sites. 

•  Daily  situation  reports  (SITREPS)  provided  to  a  large 
audience  are  very  valuable.  Need  public  and  private 
views. 

•  Time  Ordered  Event  List  (TOEL)  for  Pre-Start 
configuration  proved  very  beneficial  in  starting  the 
comp  icated  environment  in  a  structured,  repeatable 
manner 

•  Having  a  procedure  for  data  transfer  across  security 
domains  would  expedite  the  SITREPS  and  reporting 
process. 
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Spiral  Integration  Activities 


•  Multiple  spirals  are  required  to  integrate  a  large  multi-site  test  event. 

•  Complicated  operational  scenarios  with  complex  tactical 
communications  require  more  spirals.  Several  spirals  need  to  be 
devoted  to  working  operational  issues.  These  can  not  be  performed 
until  the  technical  integration  is  complete. 

•  If  technical  personnel  are  required  to  play  an  operational  role  or  use 
operational  equipment,  at  least  one  spiral  needs  to  be  devoted  to  this 
training. 

•  Full  integration  testing  can  not  begin  until  all  sites  are  online. 

•  It  is  possible  to  perform  multiple  activities  simultaneously  during  a 
spiral.  To  be  effect  sites  need  multiple  staff  and  communication  lines 
to  participate  in  simultaneous  activities.  Each  simultaneous  activity 
needs  a  lead. 

•  The  lead  range  should  support  one-on-one  testing  as  needed  during 
early  integration  to  help  with  application  testing. 


Future  Recommendations 


•  Persistent  networks,  test  infrastructure,  and  continual  employee 
training  are  needed  along  with  periodic  integration  activities  with 
other  organizations  to  make  LVC  distributed  testing  executable  in  a 
realistic  timeframe  and  budget. 

•  Test  control  and  Network  Characterization  tools  should  have  remote 
operation  capabilities 

•  A  real-time  TENA  capable  Cross  Domain  Solution  will  allow 
visualizing  tests  and  performing  some  data  analysis  in  an  unclassified 
environment. 

•  Consider  classified  VTC  capabilities  in  the  lab  to  aid  in 
communications.  This  was  used  between  DTCC  and  GMAN  labs  in 
JBD2  and  worked  well. 

•  A  “Test  Harness”  class  of  tool  would  aid  sites  in  certifying  the  ability 
to  exchange  data  properly  before  entering  integration. 

•  Determine  proper  method  for  Joint  testing  Security  Classification 
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Future  Recommendations 

(Cont.) 


•  Develop  network  design  approach  that  will  handle  the  tactical 
IP  space  (non-routable  addresses). 

•  Integrate  a  Cross  Domain  Solution  (CDS)  for  the  classified  and 
unclassified  Wiki 

•  Use  tactical  network  and  voice  communication  simulations  Vs. 
“perfect”  connections. 

•  Consider  performing  on-site  integration  for  selected  sites. 

•  Service  Information  Assurance  (lA)  requirements  for  all 
applications  should  be  consistent 

•  Should  have  a  backup  plan  for  all  critical  applications  and  sites 
-  will  Increase  integration  activities  and  cost. 


26 


Questions? 


